Non-Expert Correction of Automatically Generated Relation Annotations
نویسندگان
چکیده
We explore a new way to collect human annotated relations in text using Amazon Mechanical Turk. Given a knowledge base of relations and a corpus, we identify sentences which mention both an entity and an attribute that have some relation in the knowledge base. Each noisy sentence/relation pair is presented to multiple turkers, who are asked whether the sentence expresses the relation. We describe a design which encourages user efficiency and aids discovery of cheating. We also present results on inter-annotator agreement.
منابع مشابه
Cheap and Fast - But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks
Human linguistic annotation is crucial for many natural language processing tasks but can be expensive and time-consuming. We explore the use of Amazon’s Mechanical Turk system, a significantly cheaper and faster method for collecting annotations from a broad base of paid non-expert contributors over the Web. We investigate five tasks: affect recognition, word similarity, recognizing textual en...
متن کاملVisualizing Semantic Table Annotations with TableMiner+
This paper describes an extension of the TableMiner system, an open source Semantic Table Interpretation system that annotates Web tables using Linked Data in an effective and efficient approach. It adds a graphical user interface to TableMiner, to facilitate the visualization and correction of automatically generated annotations. This makes TableMiner an ideal tool for the semi-automatic creat...
متن کاملA Tool for Creating and Visualizing Semantic Annotations on Relational Tables
Semantically annotating content from relational tables on the Web is a crucial task towards realizing the vision of the Semantic Web. However, there is a lack of open source, user-friendly tools to facilitate this. This paper describes an extension of the TableMiner system, an open source Semantic Table Interpretation system that automatically annotates Web tables using Linked Data in an effect...
متن کاملTowards Automatic Sign Language Annotation for the ELAN Tool
A new interface to the ELAN annotation software that can handle automatically generated annotations by a sign language recognition and translation framework is described. For evaluation and benchmarking of automatic sign language recognition, large corpora with rich annotation are needed. Such databases have generally only small vocabularies and are created for linguistic purposes, because the ...
متن کاملInferring Local ( Non - ) Aliasing and Strings for Memory Safety 1
We propose an original approach for checking memory safety of C pointer programs, by combining deductive verification and abstract interpretation techniques. The approach is modular and contextual, thanks to the use of Hoare-style annotations (preand postconditions), allowing us to verify each C function independently. Deductive verification is used to check these annotations in a sound way. Ab...
متن کامل